Flakify: A Black-Box, Language Model-Based Predictor for Flaky Tests

Software testing assures that code changes do not adversely affect existing functionality. However, a test case can be flaky, i.e., passing and failing across executions, even for the same version of source code. Flaky cases introduce overhead to software development as they lead unnecessary attempts debug production or Besides rerunning multiple times, which is time-consuming computationally expensive, flaky predicted using machine learning (ML) models, thus reducing wasted cost re-running debugging these cases. state-of-the-art ML-based predictors rely on pre-defined sets features are either project-specific, inapplicable other projects, require access code, always available engineers. Moreover, given non-deterministic behavior cases, it challenging determine complete set could potentially associated with flakiness. Therefore, in this article, we propose Flakify, black-box, language model-based predictor Flakify relies exclusively requiring (a) (black-box), (b) rerun (c) pre-define features. To end, employed CodeBERT, pre-trained model, fine-tuned predict We evaluated two publicly datasets (FlakeFlagger IDoFT) compared our technique FlakeFlagger approach, best ML-based, white-box different evaluation procedures: (1) cross-validation (2) per-project validation, prediction new projects. achieved F1-scores 79% 73% dataset respectively. Similarly, 98% 89% IDoFT validation procedures, Further, surpassed by 10 18 percentage points (pp) terms precision recall, respectively, when dataset, bound unnecessarily percentages (corresponding reduction rates 25% 64%). also significantly higher results used suggesting better generalizability over FlakeFlagger. Our further show black-box viable option predicting